Existing Criticisms of Simulation Practices

Several recent studies have underlined the lack of rigor in simulation practices that threatens the credibility of the published claims. While agreeing with some of the criticism, this article argues against some others and advocates for a different overall view. In short, existing criticism is concentrated on simulation as an independent task, and advises better reporting (repeatability), more reliability, more realism, and more scenarios. This article leverages the views of different communities and benefits all researchers by presenting a common ground. Our starting position is that simulation is merely a supporting factor for new ideas. Better simulations may always be added later. The main contribution of an article may be a novel valuable protocol, novel concepts, or theoretical analysis, with simulation (if needed at all) only used to provide minimal but necessary support for the claims made, provide justification for later deeper simulation, emulation, or implementation, and/or indicate weaknesses to be addressed by further analysis and research. A single article will never solve all the problems on the road toward creating applications. For example, after a thousand articles on routing in ad hoc and sensor networks, it still remains unclear which protocol will perform well under a wide range of scenarios. It is our view that each article should be judged on its overall contribution, including the assumptions used, theory developed, new algorithms introduced, protocol details, simulation results, and relevance to an ultimate goal of staying on a path toward creating applications. We begin with a literature review of existing criticism for simulation practices, and then discuss what we believe are the main issues. We take a more general view of simulation as a support for new ideas and theories, providing a platform for their comparison with truly competing existing solutions, not merely for their validation. We also discuss the proper meaning of a proof of concept, selection of independent variables to clearly represent a variety of scenarios, and simplicity of overall design that should provide for understanding and explanation.